Computer-assisted surgical navigation is at the threshold of a revolution in surgery by extending the surgeon's capacity to visualize the underlying anatomy and by guiding positioning of instruments. A variety of innovations in computerized virtual image collection, analysis, fusion and generation are driving these advances. Advances have been made in gaming hardware, military hardware, and augmented reality by which direct pupillary projection is realized of images containing a combination of a camera view and a virtual construct or constructs. Increased numbers of procedures, such as implant placement in the hip and knee, can benefit from precise surgical navigation during the implantation. Improvements in outcomes, as by reduction in the number of revisions because of a misaligned implant, for example, would save more than enough to warrant further investment in improved surgical navigation technologies.
For example, as known in the gaming arts, a physical motion sensor (typically a three-axis accelerometer or gyrosensor, more generally “inertial sensors”) can be combined with a camera and display, enabling a first person perspective through a visual window into a virtual space on a display, as is described in U.S. Pat. No. 8,913,009 to Nintendo. Relative spatiality is achieved by defining a stationary window. Thus for example, a player may swing an actuator through the air in a virtual golf game, causing a virtual ball represented on a viewing screen to fly as if struck. Representative patent literature describing the workings of this technology includes U.S. Pat. Doc. Nos. 2012/0258796 and U.S. Pat. No. 8,100,769 to Nintendo, U.S. Pat. Nos. 6,285,379 and 8,537,231 to Philips, and related art for interactive virtual modeling hardware and software such as U.S. Pat. Doc. Nos. 2005/0026703 to Fukawa, 2009/0305785 to Microsoft, and U.S. Pat. No. 7,705,830 to Apple and U.S. Pat. No. 7,696,980 to Logitech, which disclose technologies for dispensing with keyboards in favor of haptic gesture sets and resultant representation and control of interactive processes. Depth modeling of physical objects using a structured pattern of dots generated by infrared emitters is described in U.S. Pat. Doc. No. 2014/0016113 to Microsoft and in U.S. Pat. No. 6,891,518 to Siemens.
Surgical use is known in the art. U.S. Pat. Nos. 6,787,750 and 6,919,867 to Siemens describe use of optical fiducials to measure depth and location in a surgery. In U.S. Pat. No. 6,919,867, an example is given (Col 4, line 8—Col 6 Line 42) where a surgeon is provided with a view of internal anatomical structures through a head-mounted display while operating. A correct anatomical orientation relative to the patient's body is achieved by mounting retroreflective optical beacons on the patient and around in the workspace and by employing image analysis to identify the location of the beacons. Computing means are taught for relating a coordinate system associated with the camera with a coordinate system relative to the patient's body and for tracking the camera as it moves with the head of the surgeon. However, after almost two decades of development, the resultant systems utilize cumbersome retroreflective balls that must be fastened to bones and surgical tools so that their positions can be mapped, and any images in the headset display appear superimposed on nearfield elements such as the surgeon's hands, defeating the surgeon's hand-eye coordination. As a result, most surgeons have reverted to display of the virtual images on a remote display that is accessed by looking up and away from the surgical site.
Infrared markers have also been used for dental surgery (Hassfeld, S et al. 1995. Intraoperative navigation in oral and maxillofacial surgery. Intl J Oral Max Surg 24:111-19). Correlation between CT and patient skin surfaces for guiding surgical procedures was achieved using a laser scanning system (Marmulla R and Niederdellman H. 1998. Computer-assisted bone navigation. J. Craniomaxillofac Surg 26:347-59) and later by the same group (Markerless laser registration in image-guided oral and maxillofacial surgery, J Oral Maxillofac Surg 62:845-51). However, these systems required immobilization of the patient in a reference frame device and again use a remote display to present the image synthesis so as to avoid visual illusions that are paradoxical and confusing.
All these systems rely on optical image analysis that depends on camera frame grabbers that are inoperable and blind when a needed line of sight is blocked. Optical systems are not operative when lighting is insufficient or a direct optical path to the target is obstructed or unrecognizable, such as when smeared with blood or when a surgeon's hands or a surgical instrument is blocking the view from the camera. Image analysis to recognize and triangulate optical fiducials is also computationally intensive, which can be slow or halting, and has the effect of limiting the availability of computer assisted surgical navigation systems by driving up the price and increasing system complexity.
Early computer-aided operating systems include HipNav, OrthoPilot and Praxim. Technologies of relevance have been developed by Simbionix, 3D Systems, BlueBelt Technologies, Medtronic and Siemens. But disadvantages of computer-assisted surgery remain. A major disadvantage is cost, which is generally prohibitive for many hospitals and surgery centers. Improvements have added to the cost, not reduced it. The size of the systems is also disadvantageous. Large C-arms or O-arms and windows take up space in the surgical suite, an important disadvantage in already crowded operating rooms of modern hospitals or clinics in that the equipment becomes a liability when fast action is needed and access is impaired. Additionally, another disadvantage of most surgical navigation systems in current use is the need for intraoperative computerized tomography (CT) imaging, which exposes the patient and staff to significant doses of ionizing radiation.
As applied to surgery, conventional systems generally use a collection of retroreflective spheres that serve as fiducial markers. Clusters of spheres are attached to surgical instruments so that orientation and depth can be monitored using cameras. A pattern of infrared dots is projected onto the surgical field and analysis of the centroid of each dot on spherical surface permits acquisition of the position of each fiducial. Each surgical instrument must include at least four fiducial markers for complete orientational mapping and the needed resolution of the centroids requires a fairly large tetrahedral cluster be used. Fiducial clusters may also be attached to the patient, such as by clipping the marker to an exposed bone. These reflective spheres are not useful, of course, if the optical path is blocked, as occurs frequently in surgery during the more invasive parts of the procedures.
Optics for infrared wavelengths rely on illumination outside the range of human vision, and hence have been adopted as a foundational technology. However, the technology may be better suited to inanimate objects rather than warm bodies. Dichroic mirrors and bandpass filters will not readily separate broadly emitting objects in the 700 to 1200 nm range. Surgical lamps, reflections of hot bulbs off chrome steel, and tools such as cauterizing tips may cause spurious images and add to computation time.
Binocular visors are known in the art and may be used in place of a remote display screen. However, by blinding the surgeon to all but camera generated views, the surgeon can be no more perceptive than the capacity of the system to generate a lifelike display in the visor. Surgeons wishing to rely on an unaided eye and their own hands to perform the procedure must remove the visor. The difficulty of this unsolved problem is underlined in recent literature reports (Bichlmeier C and N Navab, Virtual window for improved depth perception in medical AR; Blum T et al. 2012 Mirracle: an augmented reality magic mirror system for anatomy education. IEEE Virtual Reality).
Moreover, a difficult challenge has not been solved, that of presenting the fusion data as a virtual image that appears as the surgeon would see it in first-person perspective, dynamic and moving with the position of the physician's head and eyes so as to have a believable sense of depth, where the skin and the surgeon's hands are superimposed above the deeper structures. Advantageously, the view would appear as if the surgeon was provided with the capacity to look beneath the skin or surgical field and see underlying boney and visceral structures beneath. The surgical navigation tool would take on a compact and wearable format, such as a monocular eyepiece affixed to a headset worn to the operating room by the surgeon. In order to use this as an interactive intraoperative technique, a library store of patient imaging data must be fused with the surgeon's visual perspective of the surgical field so that a virtual fusion image is presented in correct anatomical alignment and registration. By so doing, the improved imaging modality can have relevance to and can be validated by the surgeon's inherent sense of spatial location, anatomy and general surgical know-to-do derived from years of visual, tactile and kinesthetic sensory experience. The imaging modality thereby would also avoid a need for cumbersome patient registration frames and remote display systems.
Also desirable is a system enabled to segregate elements of the visual field. In a first embodiment, segregation is done to identify individual bones in a dataset derived from tomography or from an AP and Lateral view by X-ray. The individual bones or clusters of bones may then be projected into a synthetic virtual view according to their surgical relevance. It then becomes possible to isolate the bones from the patient and to do more detailed analysis of structure of individual bones and functional interactions between small sets of bones. Segmentation also includes computer power to isolate visual elements such as the hands and fingers of the surgeon, surgical tools and prosthetics while reducing virtual clutter. Surprisingly, when this is done, any relevant virtual elements of the patient's anatomy and a virtual database segmenting the surgeon's hands may be operated cooperatively to show the hands occluding the virtual anatomy—or a virtual pair of hands operating in an enhanced virtual space. These and other inventive systems have not been realized in the art and are an object of the invention and is difficult or impossible to achieve using light-based image analysis and optical fiducials at any wavelength.
Thus, there is a need in the art for an intraoperative three-dimensional virtual viewing system that overcomes the above challenges, is perceptually integrated into the surgeon's view of the operation in progress, includes both haptic and pre-haptic interfaces, and overcomes system blindness when line-of-sight is blocked. Depth-enhanced virtual views of any surgical instruments and prosthetics manipulated by the surgeon are also desirable for making measurements of angles and guidepaths on instrumental approach to a surgical target, such as in implantation of surgical fixators or replacement joints, for example. A novel approach to these and other issues facing modern surgery is described that surprisingly is computationally simple and fast and has been enhanced to rely on the surgeon's touch and gestures as well as virtual image display, thus providing essentially a multi-sensorial extension of the surgeon's senses in integrated computer-assisted surgical navigation systems and methods.